Information processing apparatus and information processing method

ABSTRACT

An information processing apparatus to link devices by recognizing the appearance of a device, such as an unknown application or a dynamically changing application. The information processing apparatus according to the present disclosure includes an appearance information acquisition unit configured to acquire appearance information indicating a feature of appearance of an own device; and a sending unit configured to send the appearance information to communicate with another device that has imaged the appearance of the own device. According to such a configuration, it is possible to link devices by recognizing the appearance of a device, such as an unknown application or a dynamically changing application.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation application of U.S. patentapplication Ser. No. 15/559,890, filed Sep. 20, 2017, which is anational stage entry of PCT/JP2016/056874, filed Mar. 4, 2016, whichclaims priority from prior Japanese Priority Patent Application JP2015-073745 filed in the Japan Patent Office on Mar. 31, 2015, theentire contents of which are hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates to an information processing apparatus,an information processing method, and a program.

BACKGROUND ART

Technology that augments the real environment obtained through a cameraor like using a computer is being studied as augmented reality (AR). Inparticular, many AR applications that recognize an object and displayappropriate information in a superimposed manner, by holding a camera ofa mobile terminal over the object are being developed due to mobileterminals equipped with cameras being easier to use as the result of thepopularization of smartphones in recent years. Thus, Patent Literature 1below describes technology that assumes a program itself functions as anauthentication key.

CITATION LIST Patent Literature

Patent Literature 1:

JP 2002-344444A

DISCLOSURE OF INVENTION Technical Problem

In an AR application, it was necessary to add a special tag image ormarker image to an object that serves as the subject, to perform objectrecognition with an image obtained from a camera. Therefore, amarkerless AR method that recognizes an object by analyzing featuresobtained from the image, without using a marker image, is alsoconceivable.

However, with either approach, it was necessary to record the featuresto be recognized as dictionary data beforehand, to recognize the object.Therefore, objects for which it is difficult to acquire featuresbeforehand, such as unknown applications running on another terminal,and applications in which the state of a screen dynamically changes,were difficult to use as objects to be recognized.

Thus, there has been a desire to link devices by recognizing theappearance of a device, such as an unknown application or a dynamicallychanging application.

Solution to Problem

According to the present disclosure, there is provided an informationprocessing apparatus including: an appearance information acquisitionunit configured to acquire appearance information indicating a featureof appearance of an own device; and a sending unit configured to sendthe appearance information to communicate with another device that hasimaged the appearance of the own device.

The information processing apparatus may include a display screen. Theappearance information acquisition unit may include an image informationgeneration unit configured to generate image information indicating afeature of a screen displayed on the display screen as the appearanceinformation.

In addition, the sending unit may send, together with the imageinformation, communication information for communicating with the otherdevice.

In addition, a plurality of applications may be displayed on the displayscreen, the image information generation unit may generate the imageinformation for each of the plurality of applications, and the sendingunit may send the image information generated for each application.

In addition, communication may be performed with the other device forwhich it has determined that a captured image of the display screen andthe image information match.

In addition, the information processing apparatus may include anidentification information acquisition unit configured to acquireidentification information for identifying the other device. The sendingunit may send the identification information together with the imageinformation.

In addition, the identification information may include at least aportion of an IP address of the other device.

In addition, the identification information acquisition unit may acquirethe identification information sent by beacon, sound, or light.

In addition, the information processing apparatus may include a positioninformation acquisition unit configured to acquire position information.The sending unit may send the position information together with theimage information.

In addition, according to the present disclosure, there is provided aninformation processing method including: acquiring appearanceinformation indicating a feature of appearance of an own device; andsending the appearance information to communicate with another devicethat has imaged the appearance of the own device.

In addition, according to the present disclosure, there is provided aprogram for causing a computer to function as means for acquiringappearance information indicating a feature of appearance of an owndevice, and means for sending the appearance information to communicatewith another device that has imaged the appearance of the own device.

Advantageous Effects of Invention

As described above, according to the present disclosure, it is possibleto link devices by recognizing the appearance of a device, such as anunknown application or a dynamically changing application. Note that theeffects described above are not necessarily limitative. With or in theplace of the above effects, there may be achieved any one of the effectsdescribed in this specification or other effects that may be graspedfrom this specification.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is a schematic view of an outline of a system according to anembodiment of the present disclosure.

FIG. 1B is a schematic view of an outline of a system according to anembodiment of the present disclosure.

FIG. 2 is a schematic view of the configuration of the system accordingto the embodiment.

FIG. 3 is a flowchart illustrating the processes of generating andrecording image information by a terminal to be recognized.

FIG. 4 is a flowchart for explaining the process of image recognition bya recognizing terminal.

FIG. 5 is a flowchart for explaining the process of a dictionary datastorage function of a server.

FIG. 6 is a schematic view of examples of communication information.

FIG. 7 is a schematic view of a system in which a tabletop interactivesystem and a terminal such as a smartphone are linked.

FIG. 8 is an explanatory view illustrating a functional configurationexample of an information processing system in FIG. 7.

FIG. 9 is a schematic view of an example of linking a stand-alonedisplay to a wearable device.

FIG. 10 is a schematic view illustrating a case in which applications ona large screen display installed on a wall are recognized.

FIG. 11 is a schematic view of objects such as home electric appliancesthat are connected to a network at home.

MODE(S) FOR CARRYING OUT THE INVENTION

Hereinafter, (a) preferred embodiment(s) of the present disclosure willbe described in detail with reference to the appended drawings. In thisspecification and the appended drawings, structural elements that havesubstantially the same function and structure are denoted with the samereference numerals, and repeated explanation of these structuralelements is omitted.

Note that the description will be given in the following order.

1. Configuration example of the system

2. Terminal and server processes

3. Examples of communication information

4. Case in which application screen is a moving image

5. Narrowing down recorded information

6. Examples of application of the embodiment

6.1. Application to a tabletop interactive system

6.2. Wearable devices and other display devices

6.3. Recognition of applications on a large screen display installed ona wall

6.4. Wearable cameras and home electric appliances

1. Configuration Example of the System

FIG. 1B and FIG. 1A are schematic views of an outline of a systemaccording to an embodiment of the present disclosure. This systemperforms communication between a recognized terminal and a recognizingterminal that are linked by one terminal recognizing, via a cameraimage, an application running on another terminal, when linking aplurality of terminals. As illustrated in FIG. 1A, in this system, amobile device that serves as a recognizing terminal 200 images anapplication screen 110 of a tablet terminal that serves as a terminal100 to be recognized.

As illustrated in FIG. 1B, the terminal 100 to be recognizedsequentially records the application screen 110 as dictionary data on aserver 300. The recognizing terminal 200 acquires the dictionary datafrom the server 300 and compares the dictionary data with an imageobtained through the imaging. Then, if the result of the comparison issuch that the dictionary data and the image match, the terminal 100 andthe terminal 200 start to communicate.

FIG. 2 is a schematic view of the configuration of a system 1000according to the embodiment. As illustrated in FIG. 2, the system 1000includes the terminal 100 to be recognized, the recognizing terminal200, and the server 300. The terminal 100 to be recognized displays anapplication or the like on a display screen. The terminal 100 to berecognized also includes an image information generation unit 102, whichgenerates the application screen on the terminal 100 as imageinformation (dictionary data). The terminal 100 records the imageinformation on the server 300 by sending the image information from acommunication unit 104 to a communication unit 304 of the server 300.The server 300 has a storage unit 302 for image information. Note thatconstituent elements of the terminals 100 and 200, and the server 300,illustrated in FIGS. 1A and 1B may be formed by hardware (circuits), orby a central processing unit such as a CPU, and a program (software)that makes the central processing unit function. In this case, theprogram can be stored on a recording medium such as memory providedinside a device, or memory connected from the outside.

The image information generation unit 102 of the terminal 100 generatesimage information regarding features for image recognition and asnapshot of an application being displayed by the terminal 100, andrecords the data on a storage unit 302 of the server 300. If a pluralityof application screens are being displayed on the terminal 100, imageinformation is sent for each application screen. The terminal 100simultaneously records, on the server 300, both communicationinformation for the terminal 100 and the terminal 200 to communicatewith each other, and communication information for the terminal 200 tocommunicate with an application being displayed on the terminal 100. Theimage information can also be generated on the server 300 using snapshotdata, instead of being generated on the terminal 100 to be recognized.The application screen may be a still image or a moving image.

The recognizing terminal 200 has a camera 202, an image recognition unit204, and a communication unit 206. The camera 202 images the applicationscreen displayed on the terminal 100, and the image recognition unit 204recognizes an image obtained through the imaging. The communication unit206 communicates with the communication unit 304 of the server 300, andacquires the image information stored on the storage unit 302. The imagerecognition unit 204 recognizes the application being displayed on theterminal 100, by comparing the image information with the image datainput from the camera 202. Then, if the application screen imaged by thecamera 202 matches the image information acquired from the server 300,information for communicating with the terminal 100 and information forcommunicating with the application being displayed on the terminal 100is acquired from the storage unit 302 of the server 300, andcommunication with the terminal 100 to be recognized starts.

The storage unit 302 of the server 300 stores image information to berecorded from the terminal 100 to be recognized, information for theterminal 100 and the terminal 200 to communicate with each other, andinformation for the terminal 200 to communicate with the application ofthe terminal 100, and provides the stored data in response to a requestfrom the image recognition unit 202 of the recognizing terminal 200.Note that a dictionary data storage function of the server 300 may beconfigured on the terminal 100 having a dictionary data generatingfunction, or on the terminal 200 having an image recognizing function.

Therefore, according to the system of the embodiment, the recognizingterminal 200 and terminal 100 to be recognized can be linked by therecognizing terminal 200 recognizing, via a camera image, an applicationrunning on the terminal 100 to be recognized, when linking a pluralityof terminals.

The terminal 100 that is running the application screen to be recognizedsends the image information such as the features for image recognitionand the snapshot of the application screen to the server 300 in realtime in accordance with a change of the screen. An unknown applicationor an application with a dynamically changing state can then beidentified by image recognition, without generating and recordingdictionary data beforehand, by the recognizing terminal 200 comparingthis image information with the image from the camera 202.

As a precondition when carrying out the embodiment, the terminal 100 tobe recognized, the recognizing terminal 200, and the server 300 areconnected beforehand by a network or P2P so as to be able to communicatewith one another. The method of connection is not particularly limited,however.

2. Terminal and Server Processes

Next, the processes of generating and recording the image information bythe terminal 100 to be recognized will be described with reference toFIG. 3. First, when an application on the terminal 100 to be recognizedis launched, it is determined in step S10 whether the display screen haschanged. If the display screen has changed, the process proceeds on tostep S12, where image information regarding the features for imagerecognition and the snapshot of the display screen is generated. On theother hand, if there is no change in the display screen, the processwaits for a certain period of time in step S19, and then returns to stepS10, where it is again determined whether the display screen haschanged.

After step S12, the process proceeds on to step S14, where the server300 records the snapshot and the features for image recognitiongenerated in step S12. Communication information is also recorded atthis time. The communication information is information for the terminal200 to communicate with the terminal 100, and information for theterminal 200 to communicate with the application of the terminal 100.

In the next step, step S16, it is determined whether the function of theapplication has ended. If the function of the application has ended, theserver 300 is notified of this, and the process proceeds on to step S18.In step S18, the data recorded on the server 300 is erased. After stepS18, the process ends.

Also, if in step S16 the function of the application has not ended, theprocess waits a certain period of time in step S19 and then returns tostep S10, and the processes thereafter are performed again.

Next, the process of image recognition by the recognizing terminal 200will be described with reference to FIG. 4. First, when an applicationon the recognizing terminal 200 is launched, it is determined in stepS20 whether it is necessary to recognize the terminal 100 to berecognized, to communicate with the terminal 100 to be recognized. If itis necessary to recognize the terminal 100 to be recognized, the processproceeds on to step S22, where the image information regarding the imagefeatures and the snapshot are acquired from the server 300. On the otherhand, if it is not necessary to recognize the terminal 100 to berecognized, the process waits a certain period of time in step S32 andthen returns to step S20, and the processes thereafter are performedagain.

After step S22, the process proceeds on to step S24, where the imageinput from the camera 202 is compared with the image informationacquired in step S22. If the result of the comparison is such that theimage input from the camera 202 matches the image information acquiredin step S22, the process proceeds on to step S26, and the communicationinformation stored on the server 300 is acquired. On the other hand, ifthe image input from the camera 202 does not match the image informationacquired in step S22, the process waits a certain period of time in stepS32 and then returns to step S20, and the processes thereafter areperformed again. In this matching determination, the captured image isanalyzed, the features are extracted, and these features are comparedwith the features in the image information, by a method similar to awell-known face detection algorithm or the like, for example. Then it isdetermined whether the images match, on the basis of the degree ofcorrelation by a template matching process or the like.

After step S26, the process proceeds on to step S28. In step S28, theterminal 200 communicates with the terminal 100 on the basis of thecommunication information. As a result, the terminal 200 is able tocommunicate with the application displayed on the terminal 100. In thenext step, step S30, it is determined whether the function of theapplication has ended. If the function of the application has ended, theprocess ends.

On the other hand, if the function of the application has not ended, theprocess waits a certain period of time in step S32 and then returns tostep S20, and the processes thereafter are performed again.

Next, the process of the dictionary data storage function of the server300 will be described with reference to FIG. 5. First, when thedictionary data storage function of the server 300 starts, the processwaits for a communication request from an application on the terminal100 to be recognized or an application on the recognizing terminal 200in step S40. If there is a request to record image information from theterminal 100 to be recognized in the next step, step S42, the processproceeds on to step S44 and the image information is recorded.

Also, if there is a request in step S46 to record communicationinformation from the terminal 100, the process proceeds on to step S48.In step S48, the communication information is recorded.

Also, if there is a request from the terminal 200 in step S50 to acquireimage information, the process proceeds on to step S52. In step S52, theimage information is provided to the terminal 200.

Also, if there is a request in step S54 to acquire communicationinformation from the terminal 200, the process proceeds on to step S56.In step S56, the communication information is provided to the terminal200.

In step S58, it is determined whether the function of the applicationhas ended. If the function of the application has ended, the processends. On the other hand, if the function of the application has notended, the process returns to step S40, and the processes thereafter areperformed again.

3. Examples of Communication Information

Information such as protocol name, port number, and IP address areexamples of communication information. FIG. 6 is a schematic view ofexamples of communication information. FIG. 6 is a schematic view ofcommunication information for a device A, a device B, and a device C,respectively. The communication information is defined for each of anetwork interface, an internet layer, a transport layer, an applicationlayer, and a communication format. Information relating to the networkinterface includes WiFi, Bluetooth (registered trademark), Ethernet(registered trademark), and WiFi Direct, and the like. Informationrelating to the internet layer includes an IP address and a port number(IPv4 and IPV6). Information relating to the transport layer is TCP orUDP information. Information relating to the application layer includesHTTP, HTTPS, WebSocket (ws) and secure WebSocket (wss) and the like.Information relating to the communication format includes JSON PRC,SOAP, and REST, and the like.

The terminal 100 to be recognized and the recognizing terminal 200 areable to communicate with each other by sharing communication informationvia the server 300. The terminal 200 recognizes the terminal 100 by theIP address included in the communication information. The terminal 200also recognizes the application of the terminal 100 by the port numberincluded in the communication information. The communication informationis linked to the image information and sent from the terminal 100 to theterminal 200 for each application screen, and is stored, together withthe image information, on the storage unit 302. The image information issent, together with the linked communication information, to theterminal 200 in response to a request from the terminal 200 to acquirethe image information. Therefore, even if there are a plurality ofapplication screens on the terminal 100, the terminal 200 is able tocommunicate with the application imaged by the camera 202, among theplurality of applications, by acquiring the port number corresponding tothe image information.

4. A Case in which the Application Screen is a Moving Image

As described above, the application screen may be a moving image. If theapplication screen is a moving image, a mechanism for absorbing a timelag in the communication can be introduced. For example, a frame numbermay be sent from the terminal 100 to be recognized to the server 300before the image information. There is no time lag in the transmissionof the frame number. Time information is linked to the frame number, sothe server 300 is able to recognize in advance that image informationwill be received. The server 300 receives the image information from theterminal 100 after the frame number. Then, when the server 300 receivesthe image information from the terminal 100, the image information forthe frame corresponding to the requested time is extracted and sent tothe terminal 200 in response to the request already received from theterminal 200. As a result, the terminal 200 is able to determine whetherthe image from the camera at the requested time matches the imageinformation sent from the server 300.

Alternatively, a moving image captured by the camera 202 may also bestored (cached) for just a certain period of time in the recognizingterminal 200. As a result, even if there is a time lag when the terminal200 receives the image information from the server 300, it is possibleto determine whether the image information matches the cached movingimage by comparing the received image information with the stored movingimage, on the basis of the time information for the frame, on theterminal 200 side.

5. Narrowing Down Recorded Information

In a case where multiple terminals 100 to be recognized have recordedimage information and communication information on the server 300, theserver 300 that has received a request to acquire the image informationand communication information from the recognizing terminal 200 narrowsdown the information from among the large amount of image informationand communication information recorded, and sends the information to theterminal 200 that sent out the request to acquire the information.

In the embodiment, a search on the server 300 side can be made easier byusing supplementary information for narrowing down the information.Position information is an example of such supplementary information.The terminal 100 to be recognized sends, together with the dictionarydata, position information for the terminal 100 acquired by a positioninformation acquisition unit (GPS) 106, to the server 300. The server300 records the position information together with the image informationand the communication information.

When the recognizing terminal 200 requests image information from theserver 300, the recognizing terminal 200 sends the position informationfor the terminal 200 acquired by a position information acquisition unit(GPS) 208 to the server 300. An information extraction unit 306 of theserver 300 narrows down the image information and the communicationinformation on the basis of the position information acquired from theterminal 200. For example, the server 300 extracts image information andcommunication information for a terminal 100 positioned within a10-meter radius of the position of the terminal 200 on the basis of theposition information acquired from the terminal 200, and sends thisimage information and communication information to the terminal 200. Bynarrowing down the image information and communication information formultiple terminals 100 on the basis of position information in this way,a comparison between the image information and the imaging informationcan be easily performed on the terminal 200 side, which enables theprocessing load to be significantly reduced.

Various types of information aside from position information can be usedas the supplementary information. For example, an identificationinformation output unit 209 of the terminal 200 sends identificationinformation to the terminal 100 using Beacon Wi-Fi, sound, or light orthe like, from the recognizing terminal 200 toward the terminal to berecognized. An identification information acquisition unit 108 of theterminal 100 to be recognized acquires the identification information.The terminal 100 sends the identification information, together with theimage information and the communication information, to the server 300,and the server 300 then records this identification information,together with the image information and the communication information.

When the recognizing terminal 200 requests image information from theserver 300, the recognizing terminal 200 sends the identificationinformation to the server 300. The server 300 narrows down the imageinformation on the basis of the identification information acquired fromthe terminal 200, and then sends the image information and communicationinformation linked to identification information that matches theidentification information sent from the terminal 200, from among theimage information and communication information recorded, to theterminal 200. As a result, the terminal 200 is able to extract only theimage information for the imaged terminal 100, from the large amount ofimage information recorded. The IP address, or a portion of the IPaddress, of the terminal 200 can be used as the identificationinformation. By narrowing down the image information for multipleterminals 100 on the basis of identification information in this way, acomparison between the image information and the imaging information canbe easily performed on the terminal 200 side, which enables theprocessing load to be significantly reduced.

Also, the dictionary data may be searched in order from the mostrecently recorded, on the basis of the order in which the dictionarydata was recorded on the server 300.

6. Examples of Application of the Embodiment

6.1. Application to a Tabletop Interactive System

Several examples in which the embodiment has been applied are describedbelow. FIG. 7 is a schematic view of a system in which a tabletopinteractive system and a terminal such as a smartphone are linked. Asillustrated in FIG. 7, this system 1100 a includes an input unit 1110 aand an output unit 1130 a. The information processing system 1100 aaccording to an embodiment of the present disclosure illustrated in FIG.7 displays information on a top surface of a table 1140 a, and allows auser using the information processing system 1100 a to manipulate theinformation displayed on the table 1140 a. As illustrated in FIG. 7, themethod for displaying the information on the top surface of the table1140 a is also referred to as a “projection type”.

The input unit 1110 a is a device that inputs content of an operation bythe user using the information processing system 1100 a, and the shapeand pattern and the like of an object placed on the table 1140 a. In theexample illustrated in FIG. 7, the input unit 1110 a is provided in astate suspended from a ceiling, for example, above the table 1140 a.That is, the input unit 1110 a is provided away from the table 1140 a onwhich the information is to be displayed. A camera that images the table1140 a with a single lens, a stereo camera capable of imaging the table1140 a with two lenses and recording information in the depth direction,or a microphone for recording sounds spoken by a user using theinformation processing system 1100 a or ambient sounds of theenvironment where the information processing system 1100 a is placed, orthe like may be used as the input unit 1110 a.

If a camera that images the table 1140 a with a single lens is used asthe input unit 1110 a, the information processing system 1100 a is ableto detect an object placed on the table 1140 a, by analyzing the imagecaptured by the camera. Also, if a stereo camera is used as the inputunit 1110 a, a visible light camera or an infrared camera or the like,for example, can be used as the stereo camera. By using a stereo cameraas the input unit 1110 a, the input unit 1110 a can acquire depthinformation. By acquiring depth information with the input unit 1110 a,the information processing system 1100 a is able to detect a hand or anobject placed on the table 1140 a, for example. Also, by acquiring depthinformation with the input unit 1110 a, the information processingsystem 1100 a is able to detect when a hand of the user contacts or isclose to the table 1140 a, and detect when the hand leaves the table1140 a. Note that in the description below, movements in which the userbrings an operating body such as a hand into contact with, or close to,an information display surface will also collectively be referred tosimply as a “touch”.

Also, if a microphone is used as the input unit 1110 a, a microphonearray for picking up sounds in a specific direction can be used as themicrophone. If a microphone array is used as the input unit 1110 a, theinformation processing system 1100 a may adjust the pickup direction ofthe microphone array to a suitable direction.

Hereinafter, mainly a case in which an operation by the user is detectedfrom an image captured by the input unit 1110 a will be described, butthe present disclosure is not limited to this example. The operation bythe user may also be detected by a touch panel that detects the touch ofa finger or the like of the user. Also, aside from this, a useroperation that can be acquired by the input unit 1110 a can include astylus operation with respect to an information display surface, or agesture with respect to a camera or the like, for example.

The output unit 1130 a is a device that displays information on thetable 1140 a and outputs audio, in accordance with information input bythe input unit 1110 a, such as the content of an operation by the userusing the information processing system 1100 a, the content ofinformation being output by the output unit 130 a, and the shape andpattern and the like of an object placed on the table 1140 a. Aprojector or a speaker or the like, for example, is used as the outputunit 1130 a. In the example illustrated in FIG. 7, the output unit 1130a is provided in a state suspended from a ceiling, for example, abovethe table 140 a. If the output unit 1130 a is configured by a projector,the output unit 1130 a projects information onto the top surface of thetable 1140 a. If the output unit 1130 a is configured by a speaker, theoutput unit 1130 a outputs audio on the basis of an audio signal. If theoutput unit 1130 a is configured by a speaker, the number of speakersmay be one or a plurality. If the output unit 1130 a is configured by aplurality of speakers, the information processing system 1100 a maylimit the speakers from which audio is output, or may adjust thedirection in which the audio is output.

Also, if the information processing system 1100 a is a projection typesystem as illustrated in FIG. 7, the output unit 1130 a may also includelighting equipment. If the output unit 1130 a includes lightingequipment, the information processing system 1100 a may control theon/off state and the like of the lighting equipment on the basis ofinformation input by the input unit 1110 a.

The user using the information processing system 1100 a is able tomanipulate the information displayed on the table 1140 a by the outputunit 1130 a, by placing a finger or the like on the table 1140 a. Also,by placing an object on the table 1140 a and having the input unit 1110a recognize the object, the user using the information processing system1100 a is able to execute various operations relating to the recognizedobject.

Note that, although not illustrated in FIG. 7, another device may beconnected to the information processing system 1100 a. For example,lighting equipment for illuminating the table 1140 a may be connected tothe information processing system 1100 a. By connecting lightingequipment for illuminating the table 1140 a to the informationprocessing system 1100 a, the information processing system 1100 a isable to control the lighting state of the lighting equipment inaccordance with the state of the information display surface.

FIG. 8 is an explanatory view illustrating a functional configurationexample of an information processing system 1100 in FIG. 7. Below, afunctional configuration example of an image processing system accordingto an embodiment of the present disclosure will be described withreference to FIG. 8.

As illustrated in FIG. 8, the information processing system 1100according to an embodiment of the present disclosure includes an inputunit 1110 a control unit 1120, and an output unit 1130.

The input unit 1110 inputs content of an operation with respect to theinformation processing system 1100 by a user using the informationprocessing system 1100, and the shape and pattern and the like of anobject placed on a surface (e.g., the table 1140 a illustrated in FIG.7) onto which information is output by the output unit 1130. The contentof an operation with respect the information processing system 1100 by auser using the information processing system 1100 includes the contentof an operation with respect to GUI that the information processingsystem 1100 outputs onto the information display surface. Informationinput by the input unit 1110, such as the content of an operation withrespect to the information processing system 1100, and the shape andpattern and the like of the object, is sent to the control unit 1120.

If the information processing system 1100 is a projection type system,the input unit 1110 may be configured by a camera with a single lens, astereo camera with two lenses, or a microphone, or the like.

The control unit 1120 controls the various units of the informationprocessing system 1100. For example, the control unit 1120 generatesinformation to be output from the output unit 1130, using informationinput by the input unit 1110. As illustrated in FIG. 8, the control unit1120 includes a detection unit 1121 and an output control unit 1122. Thedetection unit 1121 executes a process for detecting the content of anoperation with respect to the information processing system 1100 by auser using the information processing system 1100, the content ofinformation being output by the output unit 1130, and the shape andpattern and the like of an object placed on a surface (e.g., the table1140 a illustrated in FIG. 7) onto which information is output by theoutput unit 1130. The content detected by the detection unit 1121 issent to the output control unit 1122. The output control unit 1122executes control to generate information to be output from the outputunit 1130, on the basis of the content detected by the detection unit1121. The information generated by the output control unit 1122 is sentto the output unit 1130.

For example, if the information processing system 1100 is the projectiontype system illustrated in FIG. 7, the detection unit 1121 is able todetect what portion of the GUI an operating body such as a hand of theuser touched, by a correction being made beforehand such that thecoordinates on the information display surface match the coordinateswhere the operating body such as the hand of the user touched thedisplay surface.

The control unit 1120 may also be configured by a central processingunit (CPU) or the like, for example. If the control unit 1120 isconfigured by a device such as a CPU, the device may be configured by anelectronic circuit.

Also, although not illustrated in FIG. 8, the control unit 1120 mayinclude a communication function for performing wireless communicationwith another device, and a function for controlling the operation ofanother device, e.g., lighting equipment, connected to the informationprocessing system 1100.

The output unit 1130 outputs information input by the input unit 1110,in accordance with information such as the content of an operation bythe user using the information processing system 1100, the content ofinformation being output by the output unit 1130, and the shape andpattern and the like of an object placed on a surface (e.g., the table1140 a illustrated in FIG. 7) onto which the output unit 1130 outputsinformation. The output unit 1130 outputs the information on the basisof the information generated by the output control unit 1122. Theinformation output by the output unit 1130 includes information to bedisplayed on the information display surface, and audio to be outputfrom a speaker (not shown) or the like, and so on.

The information processing system 1100 illustrated in FIG. 8 may beconfigured as a single device, or a portion of the informationprocessing system 1100 or the entire information processing system 1100illustrated in FIG. 8 may be configured by separate devices. Forexample, in the functional configuration example of the informationprocessing system 1100 illustrated in FIG. 8, the control unit 1120 maybe provided in a device such as a server that is connected to the inputunit 1110 and the output unit 1130 by a network or the like. In the casewhere the control unit 1120 is provided in a device such as a server,information from the input unit 1110 is sent to the device such as theserver over the network or the like. The control unit 1120 thenprocesses the information from the input unit 1110, and information tobe output by the output unit 1130 is sent from the device such as theserver to the output unit 1130 over the network or the like.

If the information processing system 1100 according to an embodiment ofthe present disclosure is configured to project information onto atable, and enable a user to manipulate the information, as illustratedin FIG. 7, for example, the information processing system 1100 can belinked to a mobile terminal such as a smartphone on the table. Forexample, the information processing system 100 according to an exampleof the present disclosure is able to identify a mobile terminal such asa smartphone, and link to the identified mobile terminal, by the userplacing the mobile terminal on the table and having the input unit 1110recognize the mobile terminal.

However, if a plurality of users owning the exact same mobile terminalplace these same mobile terminals separately on the table at the sametime, and try to have the information processing system 1100 recognizethese mobile terminals, the information processing system 1100 will beunable to determine which of the mobile terminals to link to.

Therefore, with an embodiment of the present disclosure, even if aplurality of users owning the exact same mobile terminal place thesesame mobile terminals separately on the table at the same time, it ispossible to easily determine which mobile terminal to link to by makingthe determination using the image information described above. In thiscase, the terminal 100 to be recognized corresponds to the mobileterminal, and the recognizing terminal 100 corresponds to theinformation processing system 1100. Therefore, the informationprocessing system 1100 can be linked to each of the mobile terminals.

6.2. Wearable Devices and Other Display Devices

FIG. 9 is a schematic view of an example in which a stand-alone display400 and a wearable device 450 are linked. Here, the stand-alone display400 corresponds to the terminal 100 to be recognized, and the wearabledevice 450 corresponds to the recognizing terminal 200. The wearabledevice 450 images one application screen 410, 420, or 430 displayed onthe stand-alone display 400 using the camera 202, and compares the imageinformation recorded on the server 300 beforehand with the imaginginformation. If, upon this comparison, the image information recorded onthe server 300 beforehand and the imaging information match, thewearable device 450 is able to communicate with the application.

6.3. Recognition of Applications on a Large Screen Display Installed ona Wall

FIG. 10 is a schematic view illustrating a case in which applications ona large screen display 500 installed on a wall are recognized. Asillustrated in FIG. 10, the large screen display 500 is installed with ascreen 502 vertical to the ground. A plurality of applications 510, 520,and 530 are running on the screen 502.

Image information for each application, or an arbitrary one or aplurality of applications, displayed on the screen 502 of the largescreen display 500 is sent, together with communication information, tothe server 300 and recorded on the server 300.

On the other hand, the user uses an application on his or her smartphone600 and images the application screen displayed on the screen 502. As aresult, the smartphone 600 recognizes the screens of the applications510, 520, and 530.

The smartphone 600 corresponds to the recognizing terminal 200 describedabove. The smartphone 600 compares the image information for theapplications 510, 520, and 530 recorded on the server 300 with thecaptured image. If, upon this comparison, the image information for theapplications 510, 520, and 530 recorded on the server 300 and thecaptured image match, communication between the smartphone 600 and theapplication 510 is realized.

Various linked applications can be executed by using communicationobtained by the smartphone 600 recognizing the application screen. Forexample, image, video, and music data on the smartphone 600 can beplayed on the application 510 of the large screen display 500. Also, aplurality of users can also play card games and the like by smartphonesowned by the plurality of users recognizing one application 510 on thelarge screen display 500 and communicating with each other.

Note that in FIG. 10, the applications 510, 520, and 530 on the largescreen display 500 are recognized, but an application on a screen of thesmartphone 600 of the user can also be recognized by a camera placed onthe large screen display 500. In this case, the large screen display 500corresponds to the recognizing terminal 200, and the smartphone 600corresponds to the terminal 100 to be recognized.

6.4. Wearable Cameras and Home Electric Appliances

FIG. 11 is a schematic view of objects 700 such as home electricappliances that are connected to a network at home. These objects 700that are connected to the network correspond to the terminal 100 to berecognized. The objects 700 such as home electric appliances recordpictures of the appearance and 3D model data of themselves in thedictionary data storage function of the server 300.

Therefore, with the system illustrated in FIG. 11, the objects 700corresponding to the terminal 100 to be recognized acquire appearanceinformation relating to appearance features of themselves, and recordthis appearance information on the server 300. The user wears a wearabledevice 450 similar to the wearable device in FIG. 9. This wearabledevice 450 corresponds to the recognizing terminal 100. The wearabledevice 450 acquires images of these objects 700 by imaging the objects700 with the camera 202, and determines whether the images matches theappearance information provided by the server 300. If the images matchthe appearance information provided by the server 300, the wearabledevice 450 communicates with the objects 700.

Various applications can be executed using communication obtained byrecognition. For example, an application for setting an air conditionercan be executed by an operation from the wearable device 450, as aresult of recognizing the air conditioner. Also, an application forunlocking a lock in a door knob can be executed by an operation from thewearable device 450, as a result of recognizing the lock. Note that inFIG. 10, recognition is performed by the wearable device 450, butrecognition may also be performed by a mobile device such as asmartphone. Also, the device connected to the network in FIG. 10 is anexample, and is not limited to being a connected device or object.

As described above, according to the embodiment, an unknown applicationor a dynamically changing application can be recognized, via imagerecognition, by the sending, in real time, features and a snapshot ofthe application that is to be recognized, and using the features and thesnapshot as dictionary data in the terminal 100 that performs therecognition.

Also, a linking application using a plurality of devices can also beexecuted by being recognized by the plurality of devices. Also, when adevice or an object is connected to a network, the device or object canbe recognized by, and linked to, another device without performing therecording operation beforehand, by dynamically recording an image of theappearance, and 3D model data, of the device or object as dictionarydata.

The preferred embodiment(s) of the present disclosure has/have beendescribed above with reference to the accompanying drawings, whilst thepresent disclosure is not limited to the above examples. A personskilled in the art may find various alterations and modifications withinthe scope of the appended claims, and it should be understood that theywill naturally come under the technical scope of the present disclosure.

Further, the effects described in this specification are merelyillustrative or exemplified effects, and are not limitative. That is,with or in the place of the above effects, the technology according tothe present disclosure may achieve other effects that are clear to thoseskilled in the art from the description of this specification.Additionally, the present technology may also be configured as below.

(1)

An information processing apparatus including:

an appearance information acquisition unit configured to acquireappearance information indicating a feature of appearance of an owndevice; and

a sending unit configured to send the appearance information tocommunicate with another device that has imaged the appearance of theown device.

(2)

The information processing apparatus according to (1), including:

a display screen,

in which the appearance information acquisition unit includes an imageinformation generation unit configured to generate image informationindicating a feature of a screen displayed on the display screen as theappearance information.

(3)

The information processing apparatus according to (2), in which thesending unit sends, together with the image information, communicationinformation for communicating with the other device.

(4)

The information processing apparatus according to (2), in which

a plurality of applications are displayed on the display screen,

the image information generation unit generates the image informationfor each of the plurality of applications, and

the sending unit sends the image information generated for eachapplication.

(5)

The information processing apparatus according to (2), in whichcommunication is performed with the other device for which it hasdetermined that a captured image of the display screen and the imageinformation match.

(6)

The information processing apparatus according to (2), including:

an identification information acquisition unit configured to acquireidentification information for identifying the other device,

in which the sending unit sends the identification information togetherwith the image information.

(7)

The information processing apparatus according to (6), in which theidentification information includes at least a portion of an IP addressof the other device.

(8)

The information processing apparatus according to (6), in which theidentification information acquisition unit acquires the identificationinformation sent by beacon, sound, or light.

(9)

The information processing apparatus according to (2), including:

a position information acquisition unit configured to acquire positioninformation,

in which the sending unit sends the position information together withthe image information.

(10)

An information processing method including:

acquiring appearance information indicating a feature of appearance ofan own device; and

sending the appearance information to communicate with another devicethat has imaged the appearance of the own device.

(11)

A program for causing a computer to function as

means for acquiring appearance information indicating a feature ofappearance of an own device, and

means for sending the appearance information to communicate with anotherdevice that has imaged the appearance of the own device.

(12)

An information processing apparatus including:

an imaging unit configured to image another device;

an appearance information acquisition unit configured to acquireappearance information indicating a feature of appearance of the otherdevice from a server;

an image recognition unit configured to compare the captured imageobtained through the imaging performed by the imaging unit with theappearance information; and

a communication unit configured to communicate with the other device ifthe result of the comparison by the image recognition unit is such thatthe captured image obtained through the imaging performed by the imagingunit and the appearance information match.

(13)

An information processing apparatus including:

an appearance information acquisition unit configured to acquireappearance information indicating a feature of appearance of a firstterminal from the first terminal;

a storage unit configured to store the appearance information; and

a sending unit configured to send, in response to a request from asecond terminal, the appearance information to the second terminal tocause the second terminal to compare imaging information obtained byimaging appearance of the first terminal with the appearanceinformation.

REFERENCE SIGNS LIST

-   100 terminal to be recognized-   102 image information generation unit-   104 communication unit-   106 GPS-   108 identification information acquisition unit-   200 recognizing terminal

The invention claimed is:
 1. A communication processing apparatus,comprising: an imaging device configured to acquire captured-imageinformation; and circuitry configured to: acquire, from a server, a partof displayed-image information representing a specific application,wherein the part of the displayed-image information is displayed and isdynamically changed by a display device different from the communicationprocessing apparatus; determine, based on image recognition, whether thecaptured-image information matches the part of the displayed-imageinformation; acquire, from the server, communication information basedon the determination that the captured-image information matches thepart of the displayed-image information, wherein the communicationinformation is defined for at least one of a network interface, aninternet layer, a transport layer, an application layer, or acommunication format, and allows the communication processing apparatusto communicate with the display device or the specific application; andinitiate communication with the display device or the specificapplication based on the communication information.
 2. The communicationprocessing apparatus according to claim 1, wherein the part of thedisplayed-image information includes a part of a moving image of thespecific application, and wherein the circuitry is further configuredto: send, to the server, a request to send the part of the moving image;and acquire, from the server, the part of the moving image of which adisplayed timing corresponds to a timing of the request.
 3. Thecommunication processing apparatus according to claim 1, furthercomprising a position sensor configured to acquire position informationof the communication processing apparatus, wherein the circuitry isfurther configured to: send, to the server, a request to send the partof the displayed-image information along with the position information;and acquire, from the server, the part of the displayed-imageinformation corresponding to the position information.
 4. Thecommunication processing apparatus according to claim 1, wherein thespecific application is an unknown application for the communicationprocessing apparatus, and wherein the communication processing apparatusdoes not have dictionary data of the specific application for the imagerecognition before acquiring the part of the displayed-imageinformation.
 5. The communication processing apparatus according toclaim 1, further comprising a storage unit including at least one ofimage data, video data, or music data, wherein the circuitry is furtherconfigured to control the display device to play the at least one of theimage data, the video data, or the music data based on the communicationinformation.
 6. A communication processing method, comprising acquiring,from a first device, a part of displayed-image information representinga specific application, wherein the part of the displayed-imageinformation is displayed and is dynamically changed by the first device;acquiring, from an imaging device of a second device different from thefirst device, captured-image information representing surroundings ofthe second device, determining, based on image recognition, whether thecaptured-image information matches the part of the displayed-imageinformation; acquiring, from the first device, communication informationbased on the determination that the captured-image information matchesthe part of the displayed-image information, wherein the communicationinformation is defined for at least one of a network interface, aninternet layer, a transport layer, an application layer, or acommunication format, and allows the second device to communicate withthe first device or the specific application; and initiating, based onthe communication information, communication between the first deviceand the second device or between the specific application and the seconddevice.
 7. A non-transitory computer-readable medium, having storedthereon, computer-executable instructions, which when executed by aprocessor of a communication processing apparatus, cause the processorto execute operations, the operations comprising: acquiring, from afirst device, a part of displayed-image information representing aspecific application, wherein the part of the displayed-imageinformation is displayed and is dynamically changed by the first device;acquiring, from an imaging device of a second device different from thefirst device, captured-image information representing surroundings ofthe second device, determining, based on image recognition, whether thecaptured-image information matches the part of the displayed-imageinformation; acquiring, from the first device, communication informationbased on the determination that the captured-image information matchesthe part of the displayed-image information, wherein the communicationinformation is defined for at least one of a network interface, aninternet layer, a transport layer, an application layer, or acommunication format, and allows the second device to communicate withthe first device or the specific application; and initiating, based onthe communication information, communication between the first deviceand the second device or between the specific application and the seconddevice.